SRE-3704 ci: Fault injection testing stage on VM/bare metal by grom72 · Pull Request #17953 · daos-stack/daos

grom72 · 2026-04-09T06:02:13Z

This PR introduces logic that simplifies the Fault Injection testing stage in CI (the Jenkinsfile)
by moving it from a Docker container environment to a provisioned VM/bare metal environment.

Requires:

https://github.com/daos-do/ansible-lab/pull/153 (for KVM setup via ansible)
or
https://github.com/daos-do/toolbox/pull/28 (for manual KVM update, requires already VMs with 27GiB reserved)
https://github.com/daos-stack/system-pipeline-lib/pull/23
SRE-3704 ci: adapt NLT pipeline to cover Fault Injection testing procedure. pipeline-lib#516
SRE-3774 ci: Disable maldet on CI nodes #18203
or
https://github.com/daos-do/packer-images/pull/24

Background

The old Fault injection testing stage ran NLT fault injection inside a Docker container
(docker_runner_fi + Dockerfile.el.9) on a shared Jenkins agent host. Up to 10 FI
containers could execute simultaneously on the same host alongside other CI workloads,
resulting in severe CPU and network resource contention. The symptoms were well-documented:
RPC timeouts, SWIM protocol failures to make progress, and "Sluggish EC boundary" warnings —
all caused by infrastructure overload rather than real code defects.

Additionally, nlt_server.yaml had ABT_STACK_OVERFLOW_CHECK=mprotect set, which causes Argobots to issue mprotect() calls for ULT stack overflow detection. On KVM-based VMs, each such call triggers TLB shootdown IPIs across all vCPUs, making test execution significantly slower on VMs than on bare metal or inside Docker containers where this overhead is less pronounced. This was a known cause of very long and unpredictable FI test execution times when running on VMs. Now that the stage runs on a dedicated provisioned VM with proper resources, ABT_STACK_OVERFLOW_CHECK=mprotect is removed from nlt_server.yaml, restoring test execution duration comparable to bare metal.

Two workarounds were introduced to mask this instability:

PR DAOS-623 test: add allowed error for FI #17959 (DAOS-623 test: add allowed error for FI, commit e0fd4e3):
added skip_substrings filters in node_local_test.py and cart_logtest.py to suppress
SWIM/network-related error conditions ("sluggish ec boundary report from rank",
"sluggish stable epoch reporting", "progress callback was not called for too long",
"rpc failed; rc:") that were firing due to Docker resource contention.
PR DAOS-623 test: ignore the server errors in client FI tests too #17999 (DAOS-623 test: ignore the server errors in client FI tests too):
extended the same suppression to server-side errors seen in NLT client FI runs.

Both PRs were explicitly described as temporary workarounds, with the expectation that they
would be reverted once FI testing was moved to a dedicated, stable environment. This PR
delivers that fix and reverts both workarounds (e0fd4e3 / #17959 and #17999), restoring
full error checking in node_local_test.py and cart_logtest.py.

Solution

The NLT Fault Injection testing stage now runs on a dedicated provisioned VM
(CI_FI_1_LABEL, default ci_fi_vm1) using the same unitTest/unitTestPost pipeline
procedures as the NLT and Unit Test stages. This mirrors how NLT tests have always been
run — on bare metal/VM nodes exclusively allocated for that purpose — and brings the same
benefits to FI testing:

Dedicated CPU and memory resources (provisioned with at least 20 cores via
VM_CPUS=20 in pipeline-lib) eliminate the resource contention that caused SWIM
and RPC failures. With 20+ cores, AllocFailTest.launch() can run FI tests in
parallel (max_child = 15) instead of the forced serial mode (max_child = 1)
that occurred when the Docker container saw fewer than 20 vCPUs.
ABT_STACK_OVERFLOW_CHECK=mprotect is removed from utils/nlt_server.yaml,
eliminating the cascading TLB shootdown IPIs that occurred when multiple FI
containers ran simultaneously on a shared KVM host.
Predictable and stable execution duration — no more variance caused by competing
Docker containers on a shared host.
Pre-built RPMs are installed rather than building from source inside the container,
removing a full SCons build from the critical path and significantly reducing
stage runtime.
Full error checking is restored in node_local_test.py and cart_logtest.py;
the skip_substrings suppression introduced in DAOS-623 test: add allowed error for FI #17959 and DAOS-623 test: ignore the server errors in client FI tests too #17999 is removed.
Log collection, JUnit reporting, and issue tracking use the standard unitTestPost
path, consistent with all other test stages.

    stage('NLT Fault Injection testing') {
        agent { label params.CI_FI_1_LABEL }
        steps {
            job_step_update(
                unitTest(timeout_time: 240,
                         inst_repos: daosRepos(),
                         test_script: 'ci/unit/test_nlt.sh --memcheck no' +
                                      ' --system-ram-reserved 4 --server-debug WARN' +
                                      ' --log-usage-import nltr.json' +
                                      ' --log-usage-save nltr.xml' +
                                      ' --class-name fault-injection fi',
                         unstash_opt: true,
                         unstash_tests: false,
                         inst_rpms: unitPackages(target: 'el9') + ' daos-client-tests',
                         image_version: 'el9.7'))
        }
        post {
            always {
                unitTestPost artifacts: ['nlt_logs/'], ...
                archiveArtifacts artifacts: 'nlt_logs/fault-injection/', allowEmptyArchive: true
                job_status_update()
            }
        }
    }

The stage is renamed from Fault injection testing to NLT Fault Injection testing to
avoid confusion with the existing Fault injection testing stage and to enable detection
in parseStageInfo / skipStage in pipeline-lib.

Jenkinsfile:

Replace the Fault injection testing stage (Docker build + nlt_test()) with the new
NLT Fault Injection testing stage running on a provisioned VM via unitTest.
Remove the nlt_test() helper function entirely — its logic is now handled by
unitTest/unitTestPost in pipeline-lib.
Add the CI_FI_1_LABEL parameter (ci_fi_vm1) for the new FI VM pool; rename
CI_NLT_1_LABEL default from ci_nlt_1 to ci_nlt_vm1.
Remove the fault-inject-valgrind stash from valgrindReportPublish — FI runs
with --memcheck no and produces no memcheck XML.

ci/docker_nlt.sh:

Delete the file — the FI stage no longer builds in Docker; it runs on a provisioned VM
via the standard unitTest path.

ci/provisioning/post_provision_config_common_functions.sh:

Stop and disable maldet on provisioned nodes; maldet scans add CPU load during NLT tests.

ci/unit/test_nlt.sh:

Replace ssh -tt + inline heredoc execution with ssh -T … bash -s -- $* piping
test_nlt_node.sh over stdin, so that command-line arguments ($*) are forwarded
correctly to test_nlt_node.sh (required for the --memcheck no --class-name fault-injection fi arguments passed by the FI stage).

ci/unit/test_nlt_node.sh:

Remove sudo mkdir -p /mnt/daos (no longer needed on provisioned VMs).
Accept forwarded arguments via $*; default to the original NLT run parameters when
no arguments are given, making the script reusable for both plain NLT and FI.
Mount a tmpfs on nlt_logs/ and set TMPDIR to it before executing
node_local_test.py, so NLT log files land on a fast in-memory filesystem.
Use exec env to set HTTPS_PROXY/NO_PROXY cleanly.

ci/unit/test_nlt_post.sh:

Add a second rsync pass to also collect logs from build/nlt_logs/ on the node
(NLT with --no-root writes logs there instead of /tmp/).
Make both rsync calls non-fatal (|| true) so post steps do not fail on missing
log directories.

utils/nlt_server.yaml:

Remove ABT_STACK_OVERFLOW_CHECK=mprotect from engine env_vars; the mprotect-based
ULT stack overflow detection is no longer needed and was a source of TLB shootdown
overhead on shared KVM hosts.

utils/node_local_test.py:

Remove the skip_substrings workaround block (revert of DAOS-623 test: add allowed error for FI #17959 / e0fd4e3 and
DAOS-623 test: ignore the server errors in client FI tests too #17999): "sluggish ec boundary report from rank", "sluggish stable epoch reporting",
"progress callback was not called for too long", "rpc failed; rc:" are no longer
suppressed — these conditions should not occur on a dedicated VM.
Add fallback fault_status detection: if the initial detection fails, try fault_status
on $PATH and then /usr/bin/fault_status before giving up, improving robustness when
the binary is installed via RPM rather than built in-tree.

src/tests/ftest/cart/util/cart_logtest.py:

Remove self.skip_substrings = [] and the associated substring-suppression check block
(revert of the cart_logtest.py portion of DAOS-623 test: add allowed error for FI #17959 / DAOS-623 test: ignore the server errors in client FI tests too #17999), restoring full log
error detection.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-04-09T06:02:31Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3704

daosbuild3 · 2026-04-16T13:36:57Z

Test stage Test RPMs on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/56/execution/node/424/log

daosbuild3 · 2026-04-16T14:01:44Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/56/display/redirect

daosbuild3 · 2026-04-16T14:01:52Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/56/display/redirect

daosbuild3 · 2026-04-20T07:49:02Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/62/execution/node/369/log

daosbuild3 · 2026-04-20T14:55:19Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17953/69/testReport/

daosbuild3 · 2026-04-21T15:15:28Z

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17953/76/testReport/

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Skip-python-bandit: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Skip-func-vm: true Skip-test-el-9.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Skip-python-bandit: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-test-el-9.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Cancel-prev-build: false Skip-python-bandit: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Skip-func-test-el9: true Skip-test-el-9.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Cancel-prev-build: false Priority: 2

Run NLT and Fault Injection Tests no dedicated VMs with 64GiB of memory reserved. Limit NLT memory to 16GiB Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true

daosbuild3 · 2026-05-04T12:02:54Z

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/92/display/redirect

daosbuild3 · 2026-05-04T12:02:55Z

Test stage NLT Fault injection testing completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/92/display/redirect

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

This reverts commit 2323fd9. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Fault injection must have NLT in stage name Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

This reverts commit b03decb. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Fix access right for nlt_logs Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

mprotect-based Argobots ULT stack overflow checking causes a TLB shootdown IPI on every stack allocation/deallocation. On KVM hosts running multiple VMs in parallel this results in VM exits across all vCPUs, significantly increasing latency under concurrent load. Remove the setting to use the default (no overflow check), which is acceptable for a CI/test environment where crashes are already caught by the test harness. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

….yaml" This reverts commit dd9c9c0. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

…r.yaml" This reverts commit adcac00. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

…Test-FI Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

- Add fallback `fault_status` detection: if the primary detection via `$PREFIX/bin` fails, try resolving `fault_status` via `$PATH`, improving robustness when the binary is installed via RPM rather than built in-tree. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 4 times, most recently from 276641f to 23827b4 Compare April 13, 2026 07:37

github-advanced-security AI found potential problems Apr 13, 2026

View reviewed changes

Comment thread utils/docker/Dockerfile.el.9 Fixed

Comment thread utils/docker/Dockerfile.el.9 Fixed

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from e724b71 to 14b4ae9 Compare April 16, 2026 13:25

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 2 times, most recently from eea6d40 to 8bf0a15 Compare April 20, 2026 07:48

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from 3591870 to fdc56a7 Compare April 20, 2026 14:09

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from dd13687 to 07365a4 Compare April 21, 2026 13:00

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch from 45aa283 to 51c7487 Compare April 29, 2026 10:44

grom72 added 6 commits April 29, 2026 15:27

Remove temporary obsolete stages

1cae67d

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Run fi tests on NLT node

2b56e8c

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Cancel-prev-build: false Priority: 2

grom72 added 13 commits May 5, 2026 01:47

Ubuntu docker w/o proxy

2323fd9

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Disable maldet fo CI nodes

55294f2

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch from d1a18e4 to bdd0209 Compare May 6, 2026 17:55

grom72 added 7 commits May 7, 2026 09:54

Reapply "nlt: remove ABT_STACK_OVERFLOW_CHECK=mprotect from nlt_serve…

df9c31d

…r.yaml" This reverts commit adcac00. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Merge remote-tracking branch 'origin/master' into grom72/SRE-3704-CI-…

6cea7a1

…Test-FI Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 changed the title ~~SRE-3704 ci: CI-Test-FI~~ SRE-3704 ci: Fault injection testing stage on VM/bare metal May 8, 2026

grom72 added 2 commits May 8, 2026 14:49

grom72 requested review from janekmi and ryon-jensen May 8, 2026 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SRE-3704 ci: Fault injection testing stage on VM/bare metal#17953

SRE-3704 ci: Fault injection testing stage on VM/bare metal#17953
grom72 wants to merge 58 commits intomasterfrom
grom72/SRE-3704-CI-Test-FI

grom72 commented Apr 9, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 20, 2026

Uh oh!

daosbuild3 commented Apr 20, 2026

Uh oh!

daosbuild3 commented Apr 21, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Conversation

grom72 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Solution

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Apr 9, 2026

Uh oh!

Uh oh!

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 20, 2026

Uh oh!

daosbuild3 commented Apr 20, 2026

Uh oh!

daosbuild3 commented Apr 21, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

grom72 commented Apr 9, 2026 •

edited

Loading